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ABSTRACT 

This meta-evaluation sought to evaluate the 
personnel, procedures, and impact of a statewide evaluation system. 
Personnel evaluation provided both formative and summative 
information, as well as information pertaining to future selection < 
evaluation personnel. The evaluation of procedures led to many 
revisions and refinements in the procedures and instrumentation of 
the evaluation system. The impact study supplied data to indicate 
data that the evaluation system did have considerable impact in 
bringing about changes and improvements in local plans. Significant 
side effects included the reorganization of State Educational 
Organization (SEA) consultant services. In addition to meeting the 
informational needs for revision of the system in Illinois, the 
meta-evaluation has yielded data to justify the retention of the 
three-phase system in future years. (Author/CK) 
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Invariably when an expenditure is made, especially a large one, 
there exists a coinciding need, and often times requirement, for an 
evaluation of that expenditure. Evaluators will be the first to 
express this need; however, evaluators and their funding agents often 
fail to realize the need to assess the efficiency or merit of their 
own evaluative efforts. Worthen (1972) upholds this contention and 
states that, "only a small portion of evaluation studies are ever evalu- 
ated, even in the most perfunctory fashion." 

Scriven (1969) has termed this second-order evaluation "meta- 
evaluation." Meta-evaluation has been discussed and applied in several 
instances at the Ohio State University Evaluation Center. Guba and 
StufflebGam (1968) have .-suggested criteria for meta-evaluation. 

The purpose of the meta-evaluation described herein was to assess, 
formatively and siommatively , a state-wide system for evaluating the occu- 
pational education programs of local evaluation agenc:ies (LEAs) . 
Background — Description of Evaluation System 

The evaluation system under consideration is entitled the Three 
Phase System for Statewide Evaluation of Occupational Education Programs 
(TPS). This system was designed and field tested during the 1970-71 
school year and was fully implemented within 71 local educational agencies 
(LEAe) — secondary and post-secondary — during the 1971-72 school year. 
Evaluation was warranted by the fact that Illinois distributes over 
30 million dollars of state and federal funds annually to approximately 
600 LEAs for the operation of local vocational program^. The three goals 
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of the TPS are the: 1) improvement of programs, 2) accountability of 
state and federal funds, and 3) facilitation of state-level planning. 

The specifics of the TPS will not be presented in this paper 
(cf. Wentling and Klit, 1973, 1973b, 1972a, 1972b for further descrip- 
tion.) However, a siommary of the TPS will be presented to aid in the 
description of the meta-evaluation. 

The Illinois Evaluation System comprises three phases. The 
first phase involves the staff of LEAs , whose task is the evaluation 
and planning of a program which is well suited to their students. The 
key to Phase One is efficient, locally-directed evaluation and careful 
planning, culminating not only in the updating and upgrading of vocational 
offerings but in a planning document xor each LEA within the state. 
This local plan, which is completed annually and submitted to the State 
Education Agency (SEA) , later serves as a contractual agreement between 
the LEA and the SEA to justify and to determine funding. 

Following its preparation and siobsequent review by the LEA governing 
board, the Local Plan is forwarded to the SEA and Phase Two begins. 
During Phase Two, SEA personnel review and evaluate each Local Plan, 
recommending approval or nonapproval of the projected occupational 
program. Each local agency is measured against what the SEA staff consi- 
ders to be a reasonable estimate of that LEA*s potential within its own 
surroundings . 

Phase Three consists of two or three day on-site evaluations of a 
selected nximber of LEAs each year. During Phase Three r a team of indi- 
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viduals assesses the total occupational program of the LEA and makes 
specific suggestions for improvements For the purpose of Phase Three, a 
total program is conceived as comprising eight components, or areas of 
concern: Administrative Organization, Personnel, Objectives, Evaluation, 
Occupational Programs, Resources Utilized, Guidance Services, and Students 
Served. The evaluation team focuses on these components in their a^ialysis 
of an lea's program. 
Eva 1 u a t i on Personnel 

A team of individuals is selected from outside the LEA's district 
to gather and analyze information concerning the total occupational program 
offerings of the district • Team members are selected by the SEA from 
three groups of individuals: 1) practicing educators, 2) business, indus- 
trial, or labor representatives, and 3) former students. The evaluation 
team is headed by a team leader who has a contract with the SEA to conduct 
three on-site evaluations per year. The group of team leaders includes 
representatives of secondary, post secondary (community college), and 
university level institutions. 
On-site Visitation Procedures 

All third phase procedures are categorized into 6 stages which will 
become the focal points for the meta-evaluation . A brief description 
of each stage is warranted. 

Stage One — Previsi tation preparation includes the selection and 
notification of LEAs for visitation. ;During this phase team members 
are selected, team leaders are trained, and LEA officials are requested 
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to furnish team ir.eirbers with a copy of their Local Plan and other 
descriptive information regarding the LEA. 

Stage Tw-o--L£A personnel orie ntation involves the presentation by 
an SEA staff meinber of a 30 minute slide/tape review of the evaluation 
system. This presentation, occ:urring five weeks prior to a visit, 
describes the purpose and procedures of the evaluation. During the 
orientation session, an SEA staff memiDer administers a questionnaire 
(Preliminary Evaluation Instrument) to all staff members present. 
A similar questionnaire is also administered to a 15% sample of the 
occupational student body. The results of these questionnaires are 
mailed to each team menuoer prior to the visitation. 

Stage Three-- Team orientation . Each visit begins with the 
orientation of the team by the team leader. During the orientation 
session, the team members are introduced to their role as evaluators 
and the group reviews the previously provided information concerning 
the LEA. 

Stage Fo ur— Team exploration begins with the chief school adminis- 
trator's description of the LEA's local program, available resources, 
and plans for the future. Following the initial meeting the team begins 
its primary task--interviewing. The focus of all interviews is on 
assessing the Eight Areas of Concern; and the Team Member Handbook pro- 
vides a guide to this task. Students, occupational staff, nonoccupational 
staff, administrators, supportive personnel, the advisory committee, and 
ether community representatives are interviewed in the team's quest for 
information. 
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Stage Five — Preparation of the evaluation report requires a jcint 
effort on the part of a.11 teain merabers. The r^-.-Dort represents a con- 
sensus of the team' s evaluation of collected infonr.ation. The heart of 
the report is a written set of specific conclusions, recoinir.endations , and 
suggested actions for each of the eight areas of concern. 

Stage Six--The suiniriary conference is scheduled to occur at the 
conclusion of the report preparation stage. The purpose of this meeting 
is to allow the team leader to personally present the findings of the 
team to the LEA personnel. Upon completion of the summary conference, 
the evaluation report is forwarded to the SEA for duplication. Within 
three weeks, copies are returned to the LEA for distribution to LEA 
staff, Board of Education, and advisory committees. 

The evaluation report prepared during each phase three visit 
becomes a primary source of ideas for future planning and developmental 
activity to be carried on in Phase One. A key feature of the Illinois 
Evaluation System is the built-in follow-up of local changes via the 
assessment of subsequent local plans which are submitted annually by 
the LEA. 

As conceived for this project, the purposes of the meta-evaluation 
were to: 1) determine the performance, as perceived by various groups 
of individuals, of personnel involved in executing the evaluation system, 
2) assess the effectiveness of the procedures and instrumentation of the 
TPS, and 3) determine the impact of the TPS in terms of change and improve- 
ments in the Local Plans of LEAs. To accomplish these purposes the 
meta-evaluation was completed by conducting three related studies. 

o 

ERIC 
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STUDY ONE — EVALUATIOrC OF PERSO:e;£L 

SaKiple 

To acquire data for the evaluation of the personnel responsible 
for directing on-site visitations (team leaders), an attempt was 
made to solicit responses from all individuals who had contact with 
the evaluation system. Included in the sample were 226 administrators 
and 1,365 instructional and supportive personnel from within the 71 
LEAs which had been evaluated. In addition, input was derived from 
questionnaires administered to 299 evaluation team members and each 
of 7 regional directors for each visitation v/ithin their region, while 
others in the sanple were included only once. 
I ns_t r ume n t a ci o n 

Four questionnaires were utilized to elicit ratings of evaluation 
personnel. These questionnaires also contained items which related 
to the evaluation of procedures and instruments that were utilized in 
the TPS. Four questionnaires served dually in Study One (personnel 
evaluation) and in Study Two (procedure evaluation) . The titles of the 
four forms, their audience and their corresponding number of items were: 

FORM TR LEA TEACHERS AND SUPPORTIVE PERSONNEL — 35 items 

FORM AD LEA ADMINISTRATORS ~ 31 items 

FORM TM " EVALUATION TEAM MEMBERS — 41 items 

FORM RD STATE EMPLOYED REGIONAL DIRECTORS — 21 items 

Each questionnaire, printed on a different color paper, contained items 
of an agreement type with a scale of strongly agree, agree, disagree, and 
strongly disagree— all concluded with a r<^quest for additional comments 



and suggestions for improvement of the TPS. 

Instrument items were based on a delineation of duties or task analysis 
of team leaders-^and of each procedure of the TPS. Personnel evaluation 
items were categorized as they pertained to either the teait^. leader's 
performance or the team's performance. This categorization led to the 
establishment of 3 subscores; (i) team leader rating, (ii) team rating, 
and (iii) a combination of the two to form an overall team rating. 

Items pertaining to the procedural evaluation were categorized into 
the six stages of the TPS. The reporting of stage scores will be described 
further within the report of Study Two. 
Design and Procedure 

There were three purposes of the personnel evaluation which were met 
with the synthesis and a.-^.alysis of information gathered by a questionnaire 
survey. These were to: 1) provide formative evaluative information to 
team leaders, 2) provide the directors of the TPS with a summative evalu^ 
ation of team leaders, and 3) determine future selection criteria by con- 
ducting an investigation of the experiencial background of successful 
team leaders. 

A set of instruments, forms AD and TR, was mailed to the chief school 
administrator of each evaluated LEA. He was requested to administer the 
instruments to his staff and to return them by mail to the meta-evaluation 
staff. TKis was accomplished within four to six weeks following the 
on-site visit to the LEA. 

In addition to the survey of LEA personnel, each team member was 
mailed a copy of Form TM and a thank-you letter for serving on an 



evaluation team. Regional directors completed a copy of Form RD for 
each visitation in which they were involved. Follo\^7-up of non-'respondents 
led to a 100 percent return from LEAs and regional directors. A return 
of 89 percent was achieved for team members. 

Formative evaluation . To meet with the purpose of formative evalu- 
ation, summaries for each form of the data collection instrument were 
mailed to each team leader. Summaries included response summaries for 
each item which pertained to the team leader's or his team's performance. 
Also, subscore 1, si±iscore 2, and a total score were provided. Mean 
subscores for all team leaders combined were also given to provide a. 
norm from which a team leader could compare his performance. 

Summative evaluation . Subscores ut:.li2ed in the formative evaluation 
of personnel were also used in the summative evaluation. Team leader 
scores (subscore 1) were standardized and combined, and each team leader 
was ranked. An attempt was made to verify the low-ranking team leaders* 
poor performance. More information was collected from personal interviev/s 
with regional directors in this verification. Additionally, the completed 
questionnaires relative to each of the five poorest team leaders were 
examined to. identify any open comments which may have been made by those 
surveyed regarding the performance of the team leader. 

Investigation of Selected Criteria . A third step in the personnel 
evaluation involved the investigation of the previous qualifications and 
experience of successful team leaders. It was thought that this informa- 
tion would be useful in the future selection of team leaders. Two separate 



one-way Multivariate Analyses of Variance (MAMOVA) v;ere performed. 
The first considered the type of institution in which team leaders 
v;ere currently employed as the independent variable. This variable 
was classified by three levels: secondary schools r coinmunity colleges r 
and universities. The second analysis considered type of position 
held by team leaders as the independent variable • Type of position 
incorporated four levels: instructional personnel, vocational direc- 
tors, deans, and chief school administrators • The dependent variables 
in each analysis were the team leader subsco^es for each of the four 
forms of the questionnaire. 
Results and Discussion 

Fomative Evaluation of Personnel . The formative personnel evalu- 
ation resulted in the feedback of questionnaire responses and their 
combined subscores to team leaders. A summary was provided for each 
questionnaire form. This information was presented in the following 
format: 

FORM TM (TEAM MEMBER) FOR D. ALLEN 
SA A D SD n x 

00 08 38 54 13 3.46 1 The team leader was too domineering 

during the evaluation 

92 08 00 00 13 3.92 2 The team leader was an excellent 

leadex 

69 31 00 00 13 3.69 41 The team leader solicited ideas and 

comments from all team members 

Your Subsccre 1 = 3.37 
Your Subscore 2 = 3.33 
Your Total Score = 3»35 
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ncludes those items directly related to the team leader 
atems 1, 2f 8, 9, 10, 12, 41) 



Subscore 2 



Includes those items directly related to team meinbers 
(Items 1, 3, 4, 5, 6, 11, 14) 



Total 



Combines subscores 1 and 2 to give, an overall team rating 



Average for all team leaders combined. 



Subscore 1 = 3.32 
Subscore 2 = 3. 33 

Total 3.32 



Sample size equal to 299 



These summaries were intended to reveal to the team leader how specific 
aspects of his performance were perceived by the various groups of 
individuals. Ideally, this information would aid the team leaders in 
identifying and remedying their weaknesses. 



was conducted to identify both strengths and weaknesses of team leaders. 
In addition to the data provided by the questionnaire survey, observation 
by SEA staff and informal feedback by local personnel to the SEA via 
telephone r personal contact and letter, all contributed to decision 
making regarding the summative evaluation and subsequent retention of team 
leaders. Team leaders who were rated highly by LEA, SEA, and team members 
were utilized in the future training of team leaders via simulation 
experiences . 

Summative results also revealed weaknesses in team leader performance 
which were somewhat universal. This, in many instances, led to the consi** 
deration of specific jrkills in future team leader training sessions. 

Finally, summative information led to the elimination of three low 
ranking team leaders for the current evaluation year. 



Summative Evaluation of Personnel . Summative evaluation of personnel 
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Investigation of Selection Criteria , The first one-way Multivariate 
Analysis of Variance sought to determine if team leaders from one back- 
ground were rated higher than individuals from a different background. 
Individuals from three types of institutions were compared; secondary 
institutions, community colleges, and universities. Multivariate results 
were not significant (F =-70, p <.30); consequently no further analysis 
was performed. 

The second analysis considered the type of position held by team 
leaders (i.e., instructional, vocational director, community college dean, 
and chief school administrator) as the independent variable with the 
ratings from forms TR, AD, TM, and RD as criterion measures. This second 
MANOVA also failed to yield significance (F = 1.42, p< .12). 

These results did not reveal one type of person as superior to others 
in terms of ratings by various groups involved with the evaluation system. 

STUDY TWO — EVALUATION OF PROCEDURES 
Sample and Instrumenration 

The sample and instriiments described for Study One overlapped those 
of Study Two. In addition to Forms AD, TR, TM, and RD, a similar form, 
FORM TL, consisting of 38 items, was constructed to elicit the opinions 
of team leaders relative to the procedures and data gathering instruments 
of the TPS. A 100 percent return was achieved from the 25 team leaders. 
Design and Procedure 

In addition to providing data for personnel evaluation, the questionnaire 
survey provided input to the evaluation of procedures and data gathering 
instruments utilized in the on-site visitations. Questionnaire items were 
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written to elicit information pertaining to each of the six stages of 
Phase Three of the TPS. Results were summarized according to these stages. 

' 'isults of items pertaining to each stage were mailed to four of the 
Luain leaders; each team leader receiving the results of one stage. For 
example, team leaders A, B, C, and D received the results of items 
pertaining to Stage 1— -the orientation of LEA personnel. A number of items 
occured on more than one form. In this case the responses for each form 
were provided to team leaders. Below are examples of the two items in 
the format through which they were reported™ 



rorm 


# 


. SA 


A 


D 


SD 


OMIT 




X 


0 




TM 


5 


.33 


.57 


.06 


.02 


.02 


1, 


. 77 


.65 


THE INTERVIEW HANDBOOK WAS OF 


TL 


6 


.54 


.36 


.03 


.02 


.05 


1, 


.50 


.66 


HELP IN ORGANIZING INTERVIEWS. 


TL 


2 


.05 


.25 


. 39 


.30 


.02 


2, 


,95 


.87 


TEAM MEMBERS WHO INTERVIEWED EDU- 


TM 


39 


.00 


.12 


. 57 


.24 


.06 


3, 


,12 


.63 


CAt6rS DIDN'T USE THE INTERVIEW 


















HANDBOOK. 


RD 


3 


.00 


.30 


.58 


.11 


.01 


2. 


,81 


.62 





The results pertaining to each stage were mailed to the team leaders, 
who were instructed to analyze the information and to integrate their personal 
experiences relative to their specific stage. Four weeks subsequent to this 
mailing, a three-day workshop was held m which each team leader participated. 
Fifty percent of the time available in this team leader workshop was devoted 
to the analysis and discussion of the evaluation procedures and instrumentation. 
Six groups of team leaders were provided copies of comments which were made 
on the completed forms of questionnaires. The six groups worked independently, 
synthesizing questionnaire summaries, questionnaire comments, and their 
personal experiences, for seven hours. Small group activity culminated in 
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recommendations and suggestions for improving the components of each stage; 
these recommendations were presented to the entire group of team leaders 
and SEA staff members for consideration and discussion. 
Results and Discussion 

The data "from questionnaires which pertained to procedural stages, 
comments which were made by questionnaire respondents , and the personal 
experiences of team leaders allowed for the careful analysis of the 
evaluation systein. The outcome of thib analysis was a report, prepared 
by the team leaders, which, i.dentified both exemplary and deficient 
attributes of each stage of the evaluation process. In addition, specific 
recommendations were made relative to improving the existing procedures. 
The final report identified many needed improvements. This paper will 
present only several examples and their consequential action. 

One major need identified by the meta-evaluation was for the standardi- 
zation of procedure among visitations. It was recommended that each step 
in the visit and each of the team leader duties be specifically delineated. 
The resulting action on the part of the TPS staff v/as the development of 
a Team Leader Handbook (DVTE, 1972) which outlines each task within a 
specified time frame. 

A second major concern which was revealed by the meta-evaluation 
related to the means in which the overall rating for each area of concern 
was presented in the LEA evaluation report. This was changed for the 
1972-73 school year and the meta-evaluation study in progress for this year 
has revealed the need for further improvement. 
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Another example of deficiency in the TPS v;as the ins trijinentation 
utilized in data collection. The Preliminary Evaluation Instruments admi- 
nistered to staff and students prior to a visitation, the School and Commu- 
nity Data Form , and the Team Member Handbook ^ere all revised on the basis 
of the meta-evaluation- 

STUDY THREE — EVALUATION OF IMPACT 

Sample 

A stratified random sample of LEAs was drawn for the impact evaluation. 
A total of 40 LEAs were selected: 10 large unevaluated, 10 small unevalu- 
ated LEAs, 10 large evaluated, and 10 small evaluated. 
Procedure 

The impact study sought to assess the efficacy of the TPS in terms 
of bringing about change and improvement in the Local Plans of LEAs. 
Data collection for Study Three involved the analysis of annually-submitted 
planning documents (Local Plans) of 20 evaluated and 20 unevaluated LEAs 
for a two-year period. The 1971-72 school year plans (submitted prior to 
implementation of the TPS) and the plans of 1972-73 were reviewed concur- 
rently to identify all changes which had been made in the plans. Those 
plans were reviewed by a person knowledgeable of local plan composition 
but naive to the classification or strata from which the plans had been 
drawn. Changes were recorded, regardless of their magnitude or significance, 
onto 3x5 cards — one card for each change. Change information was later 
coded as to strata and Area of Concern to which it pertained, and was 
punched onto data processing cards. 
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The change cards were printed on a computer output form which included 
a Likert rating scale for each change. Copies of the printout were 
presented to 9 judges who were instructed to rate each change with regard 
to its importance in in^proving a local program. Judges' scores were 
summed, and a mean value was calculated for each change. These mean 
scores were subsequently utilized as item scores while categories of 
change—the original Areas of Concern— were considered subtests which 
resulted in subscores. These subscores, eight in all, became the dependent 
variables for the statistical analysis. 
Design 

It was the intention of the inoact study to determine if evaluated 
institutions reflected more significant changes in their most recent 
local plan than did unevaluated LEAs. The study was also constructed to 
determine if size of LEA had effect on the amount of change in Local 
Plans. Therefore, evaluated and unevaluated LEAs — institutions both under 
and over 600 enrollment—were analyzed via a 2 x 2 (evaluation x size) 
factorial design. Change subscores, one for .each of the eight areas of 
concern, were analyzed as dependent variables by utilizing Multivariate 
Analysis of Variance (MANOVA) . MANOVA was most expedient in that it not 
only provided the contribution of each main effect to each dependent 
variable separately, but simultaneously tested the concommittant effects 
to each dependent variable. Further, where the MANOVA illuminated signi- 
ficance, differences were investigated by fitting discriminant functions to 
the dependent variables to attribute the Multivariate differences. 
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Results and Discussion 

The 2x2 Multivariate Analysis of Variance, utilizing change 
subscores as dependent variables, led to significance ^ ox thf^ t^^'.a^ 'j^tIOH 
main effect. Multivariate, univariate, and step down analysis results 
are summarized in Table 1. 

TABLE 1 

Test of Significance Comparing Evaluated 
and Unevaluated LEAs on Change Subscores 



Dependent Variable 


Univariate F 


Step-down F 


Multivariate F 




(df =1, df = 36) 
h e 


(df, -1, df = 36) 
h e 


(df, = 8, df = 29) 
h e 


Admin . Organi zat ion 


4.95* 


4,^5* 




Personnel 


10.44* 


7.43* 




Objectives 


1.64 


.28 




Evaluation 


3.08 


.84 


3.13 


Occ. Programs 


6.32* 


3.84 


p < .01 - 


Resources Utilized 


12.80* 


2.52 




Guidance 


9.81* 


.45 




Students Served 


3.81 


1.84 





*p <.05 



LEAs which had been evaluated incorporated more significant changes 
in their Local Plan than did unevaluated LEAs (F=3.13, p < .01) . The 
variables primarily responsible for the significant EVALUATION effect were 
Administrative Organization (step down F = 4.95, p <.03) and Personnel (step 
down F = 7.43, p <.01). A discriminant function was applied to the results 
in an attempt to assess the contributors to the difference between the evalu- 
ated and unevaluated agencies. The function was: 
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+ . 39x, I 

? / "8 

or 

= .18 (Admin. Org.) + .31 (Personnel) + .01 (Objectives) 

+ .10 (Evaluation) + .41 (Occupational Programs) + -39 (Resoins:^-) 
+ .25 (Guidance) + .39j (Students Served) 

This function is based on lov; scores on objectives and evaluation ancl 
on liiigher scores for occupational programs, resources uitilized^ and stadsnts 
served. 

On the other main eiffect, SIZE, Multivariate results were not stalaisr' 
-tically significant (F = 1.79, p <.12). Table 2 summarxzes Multivariatse, 
Univariate, and step down analysis results for the SX2E effect. 

I^ABLE 2 

Comparison of Large and Small 
LEAs on Change Scores 



Dependent Variable 


Univariate F 


Step-down F 


Multivariate,, x 




(df =1, df = 36) 
h e 


(df^=l, df = 36) 
h e 


(df = 8, df-=29) 
h e^ 


AdmiTu Org,. 


2.48 


2.48 




Personnel 


6.15* 


4.73* 




Objectives 


.01 


.33 




Evaluation 


2.69 


1.75 


1.79 


Occ. Programs 


4.13* 


3.23 


p <.12 


Resources Utilized 


.16 


.59 




Guidance 


.23 


.72 




Studen1:s 


.86 


.24 





*p <.05 



he step down analysis revealed that the existing variance was 
embedded in the personnel subscore- Even though the size main effect 
was not statistically significant (p. .12), closer anajlysis of cell means 
revealed higher scores for large ov^r smaller LEAs on each dependent 
variable. The interaction of SIZE, and EVALUATION main effects was non- 
significant. 

The results of the impact study illuminated several factors important 
to the directors of the TPS. First, it has determined that the desired 
changes in plans, or intended improvements, are in evidence. Since local 
plans are, in essence, contractual agreements, it can be assumed that changes 
in plans will result in changes within operational programs of LEAs* 

The fact that little change was identified in both statement of 
objectives and in locally-directed evaluation has implications for future 
activity for the SEA. This finding, coupled with the fact that objectives 
and evaluation were lowest rated areas in LEA evaluation reports, confirms 
the need for a closer look at these areas and a possible reallocation of 
state resources^ both fiscal and personnel, toward the upgrading of these 
areas . 

The lack of significance for the SIZE effect and vhe interaction of 
size and evaluation effects are also of interest. It was hypothesized by 
TPS personnel that more change would occur in smaller LEAs due to smaller 
staffs and larger amounts of control by administrative staff members. The 
meta- evaluation findings did not uphold this hypothesis. A future follow-up 
of the LEAs will identify actual changes in programs and may prove to be 
different for large and small LEAs. 
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SUMMARY 

This meta-evaluation sought to evaluate the personnel, procedures, and 
impact of a statewide evaluation system. Personnel evaluation provided 
both formative and summative information, as well as information pertaining 
to future selection of evaluation personnel. The evaluation of procedures 
led to many revisions and refinements in the procedures and instriomentation 
of the evaluation system. The impact study supplied data to indicate that 
the evaluation system did have considerable impact in bringing about changes 
and improvements in local plans. 

Coincident to meeting the purposes of the meta-evaluation was the 
observation of many side effects of the evaluation system. Possibly one of 
the most significant outcomes of the evaluation system was the involvement of 
people — educators, business representatives, and students — in a cotmnon effort. 
This common effort was the occasion for in-service education as a discipline? 
and provided an excellent opening to dialogue between school and community. 

Other significant side effects included the reorganization of SEA 
consultant services. Current services are offered and almost automatically 
provided to LEAs which have been evaluated. This procedure differs greatly 
from the traditional system under which LEAs were required to request state 
services. Revision of pre-service and in-service education programs offered 
by institutions of higher education within the state is another of the many 
side benefits of the evaluation system. 

The investment in. a meta-evaluation has reaped many benefits for the 
SEA in Illinois. In addition to meeting the informational needs for revision 
of the system, the meta-evaluation has yielded data to justify the retention 
of the TPS in future years. 
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